Testing in AWS Account:

• Same Account CodePipeline Trunk

• Same Account CodePipeline GitFlow

• Same Account CDK Pipeline

• Cross Account CodePipeline Trunk

• Cross Account CodePipeline GitFlow

• Cross Account CDK Pipeline

• CDK Pipeline 3 Envs (CICD + Dev in same, Prod in cross)

• Trunk-based CodePipeline 3 Envs (CICD + Dev in same, Prod in cross)

Also verified Pipeline Stack(s) tagged with the same key-value pairs as before

2 thoughts to enhance this PR:

1. Currently, the bootstrapping required for multi-account pipelines can be a little tricky - you have to edit the original bootstrap command with an additional --trust to the CICD AWS Account to get pipelines working without interfering with previous data.all Environment set-up.

• We could look into creating a new cdk bootstrap stack specifying a new toolkit name and new qualifier or prefix, similar to how DDK CLI used to do in v0.5.1 (ddk constructs support this by adding bootstrap specific parameters in ddk.json ddk docs)
• But this will require additional manual work for users to run these bootstrap commands for their pipeline environments before creating pipelines - and may be prone to error

2. Should we move dataall/backend/dataall/modules/datapipelines/blueprints/ to under dataall/backend/dataall/modules/datapipelines/cdk/ directory to keep module file structure standardized across all modules

2 thoughts to enhance this PR:

1. Currently, the bootstrapping required for multi-account pipelines can be a little tricky - you have to edit the original bootstrap command with an additional `--trust` to the CICD AWS Account to get pipelines working without interfering with previous data.all Environment set-up.


* We could look into creating a new cdk bootstrap stack specifying a new toolkit name and new qualifier or prefix, similar to how DDK CLI used to do in v0.5.1 (`ddk` constructs support this by adding `bootstrap` specific parameters in `ddk.json` [ddk docs](https://awslabs.github.io/aws-ddk/release/latest/how-to/custom-bootstrap.html))

* But this will require additional manual work for users to run these bootstrap commands for their pipeline environments before creating pipelines - and may be prone to error


2. Should we move `dataall/backend/dataall/modules/datapipelines/blueprints/` to under `dataall/backend/dataall/modules/datapipelines/cdk/` directory to keep module file structure standardized across all modules

1. Maybe it is time to collect some feedback from customers and DDK experts on their recommendations. Personally I think it is better to separate the CDKToolkit stacks. That way we can 1. limit the permissions for each stack and 2. avoid overwriting trust relationships
2. Definitely, if possible

I have reviewed the code and overall it looks good. The next steps can go in the direction of reading the blueprint from a separate repository, but that's something that we will evaluate as a whole when reviewing pipelines

I moved blueprints/ to under the cdk/ parent directory and re-tested create, update, and delete with the latest changes

On bootstrapping requirements, definitely think it is something we should discuss and see how customers are currently using pipelines/ what would be best from DDK experts in this space

As is, I think this PR is good for final review unless we have a clear direction on how we want to proceed on bootstrapping pattern for pipelines (currently everything works as is so I am viewing this as an "enhancement" that could be handled separately if needed, but let me know your thoughts)

I agree on leaving bootstraping out of this PR as an enhancement to not overcomplicate this review.
I am testing the feature as is now in AWS

• CICD successfully deployed

Using same account for CICD and DEV. The following pipelines are created and executed successfully

• CDK-trunk
• CodePipeline trunk
• CodePipeline gitflow

Using one account for CICD and DEV and another for PROD. The following pipelines are created and executed successfully

• CDK-trunk
• CodePipeline trunk
• CodePipeline gitflow

Comments
As enhancement, we could change a bit the names/descriptions of the CloudFormation stacks.

• For the CDK pipeline there is no way to identify the pipeline stack other than the pipelineUri in the name. I think we can directly add a description field to the DDK CICDPipelineStack construct (Docs) in dataall/modules/datapipelines/cdk/datapipelines_cdk_pipeline.py
• For the data pipeline defined in dataall/modules/datapipelines/cdk/blueprints/data_pipeline_blueprint/app.py we could add the environment to the stack name just in case DEV and TEST are deployed in the same account f"{pipeline_name}-{environment_id}-DataallPipelineStack",

@dlpzx - added the enhancements to name/description of CFN resources:

1. CDKPipelines CFN Description - Added Stack Description with same formatting as other CFN Stacks created by data.all, Cloud formation stack of PIPELINE: {Pipeline Label}; URI: {Pipeline URI}; DESCRIPTION: {Pipeline Description}

2. DataPipeline Stacks - Added environment_id to the naming convention of stacks created via CodePipeline patterns (i.e. "{pipeline_name}-{environment_id}-DataallPipelineStack")

Tested again creating new pipelines after the above changes and tested for a trunk-based CodePipeline with 2 pipeline envs in the same account (i.e. dev and test both in 1 account)

Think ready for another round of review

Hi @noah-paige, I am doing a final review in AWS

• CICD successfully deployed

Using one account for CICD and DEV and another for PROD. The following pipelines are created and executed successfully

• CDK-trunk
• CodePipeline trunk
• CodePipeline gitflow - It fail once (see screenshot below), but when retrying it worked fine